1. The Field of the Invention
This invention relates generally to combining real world images viewed by a camera with scene elements from another source. More specifically, the present invention provides a method for calibrating a camera and lens such that when computer generated images are incorporated into a real world scene, the computer generated images are always positioned correctly with respect to real world objects.
2. The State of the Art
The state of the art of combining real world imagery with additional imagery provided by another source is a process requiring substantial precision. Typically, the other source of additional imagery is computer generated or some other type of synthetic imagery. Precision is required in the placement of synthetic imagery so that the combination appears realistic when viewed as a whole. For computer generated imagery to be properly placed within a combined scene, a frame of reference must be used in common by the real world and the computer generated imagery.
There are at least two different types of computer generated imagery which can be inserted within a scene of real world imagery photographed or filmed by a camera (where the camera is generally a source of video data). The first type of imagery is where real world objects are disposed in the foreground in front of a studio backdrop background, commonly constructed using a blue or green wall or screen. A synthetic scene is then added in place of the blue or green screen. The second type is where the real world objects appear to be surrounded by the computer generated objects. For example, an animated character is able to seemingly interact with real world characters.
It is often desirable to make live scene elements appear as if they are actually part of the background, and not just standing in front of it. In such cases, it is necessary to know, with considerable precision, enough information about the camera to generate the synthetic scene as if it were really part of the camera's filming environment. A synthetic element generator, or in this case a computer image generator, must know what the camera is "seeing" in order to accurately place the synthetic elements in the scene using a database of virtual information. For the computer image generator to know what the camera is seeing, at least seven parameters must be determined with a relatively high degree of precision. These include the orientation of the camera (also known as the pan, tilt and roll of the camera head), the camera position relative to a three dimensional coordinate axis (also referred to as the XYZ position by those skilled in the art), and the field of view (FOV, also known as the zoom setting).
Regarding the XYZ position, what is more specifically required is the XYZ position of the camera's nodal point. The nodal point is the exact point in space from which the perspective scene which the camera sees appears to be drawn. It is generally at a point along the optical axis of the zoom lens, but its precise position moves forward and back as a function of the zoom and focus settings of the lens. It may move many inches as the zoom and focus controls are operated.
To obtain the pan and tilt information, the camera head is instrumented with sensors which measure the rotation of the camera about the pan and tilt axes of the camera head. This pan and tilt information tells the computer image generator the instantaneous "look direction" of the camera. The roll of the camera head is assumed to be zero for a camera mounted on a pedestal, although that can be instrumental and accounted for as well.
Disclosed in more detail in a patent application by another inventor and filed simultaneously with this document is a method and apparatus for determining the pedestal's (and thus the camera's) XYZ position. The process utilizes a quick and simple alignment procedure utilizing triangulation to known pre-marked studio reference points. The alignment procedure is performed each time the pedestal is moved to a new location. It should be noted that the true "eyepoint" location is determined not only by the pedestal position, but also by how the camera is mounted on the pedestal, and also by the position of the camera/lens nodal point at any particular moment. What is most relevant to this document is a method and apparatus for measuring the camera/lens' nodal point position and field of view, for any setting of the zoom and focus rings on the camera lens. The process for determining the camera and lens nodal point offset and field of view as a function of the lens zoom and focus settings is referred to hereinafter as a lens calibration process.
The state of the art methods for executing the lens calibration process generally rely on very tedious measurements. Specifically, the method generally requires the use of an optical workbench, taking careful measurements, and recording data points manually. The process described above is time consuming, and requires manual measurements and manual entry of data points. Consequently, the process is prone to human error.
The state of the art process is tedious because the camera lens nodal point offset and the field of view must be determined for each zoom and focus setting of the camera and lens. Cameras and lenses to be used in a vertical set provide zoom and focus rings having instrument encoders which provide the ring settings. However, these ring settings do not directly provide the camera lens nodal point offset and the field of view. When the camera and lens are being used in real-time, the nodal point offset (and field of view) is obtained by taking the current zoom and focus ring settings, and processing them using a set of calibration data created specifically for the particular camera and lens. Therefore, each camera and lens combination has a unique calibration data set which must be determined before use. This is a one-time alignment/calibration procedure to create the calibration data set for each particular camera and lens combination. Typically, the calibration process presently takes many hours to perform.
It would be an improvement to provide a new method for measuring camera and lens properties in a lens calibration process which is more automated than the prior art. The new process should also eliminate manual entry of data, thereby reducing the chance for human error in the lens calibration process. The new calibration process should also reduce the time for creating a calibration data set for each camera and lens pair.